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Formative assessment generates feedback on students’ performance, thereby 
accelerating and improving student learning. Anecdotal evidence gathered by a 
number of evaluations has hypothesised that audio feedback may be capable of 
enhancing student learning more than other approaches. In this paper we report on 
the preliminary findings of a quasi-experimental study employing qualitative 
techniques for triangulation, conducted to evaluate the efficacy of formative audio 
feedback on student learning. We focus on the delivery of ‘voice emails’ to 
undergraduate students (n = 24) and evaluate the efficacy of such feedback in 
formative assessment and ergo students’ learning, as well as achieving a better 
understanding of students’ feedback behaviour post-delivery. The results indicate 
that audio feedback better conforms to existing models of ‘quality’ formative 
feedback, can enhance the student learning experience and can be more efficient 
in feedback delivery. Despite this, and high levels of feedback re-use by student 
participants, the audio treatment group underperformed in learning tasks when 
compared with the control group. Differences between the groups were not 
statistically significant and analyses of individual and mean learning gains across 
the treatment group provide little indication of improvements in learning. 
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Introduction 

Formative assessment has been shown to be effective in most educational scenarios 
(Black and William 1998) and its importance continues to be well recognised within 
pedagogical communities (Nicol and Macfarlane-Dick 2006). Formative assessment 
refers to assessment that is intended to generate feedback on student learning perfor¬ 
mance so as to modify learner thinking or behaviour, accelerate achievement and ulti¬ 
mately effect student learning improvements (Sadler 1998; Shute 2008). The literature 
on formative assessment suggests that it performs an important function in fostering 
‘deep learning’, thereby promoting conceptual understanding and learning at higher 
cognitive levels, and thus averting ‘surface’ approaches often associated with summa- 
tive assessment (Rushton 2005). 

Despite the clear pedagogical merits of formative assessment and feedback, few 
formative assessment opportunities are made available to students in higher education 
(Bone 2008). The reasons for this are complex but can generally be attributed to the 
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limited time lecturers have within semester-based systems to provide and deliver 
formative feedback, a problem exacerbated by staff research responsibilities and the 
increasingly large student cohorts that can dominate particular academic courses or 
modules (Yorke 2004). For ‘formative learning’ to occur and the benefits of formative 
assessment realised, feedback has to be timely, relevant and delivered to students prior 
to summative assessments. 

Recent advances in audio and Web 2.0 technologies present opportunities for 
providing a variety of audio-based learning materials to support and enhance student 
learning, most notably podcasts (Tynan and Colbran 2006; Kervin and Mantei 2008; 
Sutton-Brady et al. 2009; Middleton 2009). The use of audio technologies to deliver 
feedback of all types has also attracted attention from the learning technology commu¬ 
nity (Rotheram 2009; Bird and Spiers 2009). The potential for time efficiencies in 
feedback delivery is often cited as a possible benefit and a potential solution to the 
lack of formative assessment at higher education, particularly as most exploratory 
research appears to indicate that students tend to be favourably disposed to receiving 
audio feedback (Sipple 2007; Merry and Orsmond 2008; Rotheram 2009). Evidence 
gathered by a number of studies has also hypothesised that audio feedback may actu¬ 
ally be capable of enhancing student learning, thereby reinforcing its use (Ice et al. 
2007; Sipple 2007; Bird and Spiers 2009). However, research to date has predomi¬ 
nantly focused on student satisfaction or attitudes and therefore our current under¬ 
standing of audio feedback efficacy in student learning remains limited. 

In this paper we report on the preliminary findings of a quasi-experimental study 
employing qualitative techniques for triangulation. The research is conducted as part 
of a wider project funded to formally evaluate the use of audio technologies in deliv¬ 
ering formative feedback. It focuses on the use of audio feedback technology to 
deliver ‘voice emails’ to undergraduate students studying topics within the areas of 
business information management and web technologies. In particular, we attempt to 
evaluate the efficacy of audio email feedback in formative assessment and ergo 
students’ learning, as well as achieving a better understanding of students’ feedback 
behaviour post-delivery. 


Background 

The use of audio to provide feedback is not new. Cryer and Kaikumba (1987) 
conducted an interpretive study to investigate the use of audio-cassette tapes as a 
means of providing feedback on written work. Although a small study population 
prevented robust conclusions to be drawn, they found audio feedback to be well 
received by study participants. This was largely based on students’ perception that 
written feedback was often “too cryptic” to be followed or easily interpreted in 
subsequent learning tasks. The increased level of personalisation possible in audio 
coupled with variations in voice intonation assisted in fostering student motivation 
and interest in their learning, although some students found the lack of a written 
record for later reference to be problematic. Additional staff benefits, such as time 
savings, avoiding the ‘stress’ of drafting and structuring a written arguments, were 
also reported. 

Subsequent research, which has sought to exploit recent advances in audio tech¬ 
nology has, in general, reported findings not dissimilar to Cryer and Kaikumba 
(1987). Merry and Orsmond (2008) conducted a small study explicitly investigating 
student perceptions and use of formative audio feedback delivered as MP3 files via 
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email. Students were found to respond positively to audio feedback principally 
because it was easier to interpret than written feedback, was more personal and more 
detailed. In a wide-ranging qualitative study led by Rotheram (2009), students were 
“overwhelmingly positive” about audio feedback and appreciated its personal nature. 
Such results have been further corroborated in the literature. Ice et al. (2007) found 
audio feedback to invoke perceptions that tutors “cared”, a quality attributable to the 
personalised nature of feedback and the way in which this invoked improved student 
engagement with their learning. Similarly, other studies have found that students 
consider audio feedback to shorten the “social distance” between them and lecturers 
(Morra and Asis 2009). It is noteworthy that although the personal nature of audio 
feedback is generally found to be a positive aspect, recent exploratory research 
conducted by Fell (2009) noted that it was ineffective in enhancing the learning of 
postgraduate students owing to the lack of student-tutor dialogue. Fell’s findings 
appear to be atypical and may be attributable to the small number of participants used 
(n = 6). Nevertheless, the need for a student-tutor dialogue is a finding that warrants 
further exploration and is one that - for the purposes of this study - we attempt to 
control. 

Using audio approaches to capture greater feedback detail and to turn feedback 
around more quickly has been identified by many as a potential advantage of the 
approach. With the exception of Ice et al. (2007), who found that feedback time reduc¬ 
tions of almost 75% were possible in particular circumstances, suggestions of time 
efficiencies have been mixed and based largely on anecdotal evidence (for example, 
Cryer and Kaikumba 1987; Bird and Spiers 2009; Rotheram 2009). 

The generally positive student perception of audio feedback in much of the litera¬ 
ture appears to be attributed to the fact that it is better placed to meet the criteria of 
‘good feedback’. Gibbs and Simpson (2004) propose a series of conditions that assess¬ 
ment feedback should meet in order to maximise student learning. These conditions 
are varied but include several that are directly relevant to formative assessment, such 
as the need for feedback to be easily understood by the student, to be sufficiently 
detailed, timely and received by the student “when it still matters”. If audio feedback 
can better meet the criteria of ‘good’ formative feedback then it can be hypothesised 
that there exist better opportunities for improvements in - or increased - student learn¬ 
ing. Few researchers have attempted to measure student learning or better understand 
audio feedback efficacy. Sipple (2007) conducted a qualitative study focusing partic¬ 
ularly on student attitudes and perceptions of receiving audio feedback. Audio feed¬ 
back was found to positively influence student motivation and revision behaviour, 
self-confidence and student-tutor relationships. Flowever, Sipple concluded that audio 
feedback improved overall student learning, a ‘speculative’ conclusion based on 
improvements in revision behaviour and the possible benefits this would stimulate in 
student learning. Improving our understanding of this aspect motivates this current 
research study. 

Methodology 

Aims 

Recall that our current understanding of audio feedback efficacy in student learning 
remains limited. The aim of this study is therefore to formally evaluate the efficacy of 
audio feedback technology in formative assessment and ergo students’ learning, as well 
as achieving a better understanding of students’ feedback behaviour post-delivery. 
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Since it is an under-researched area, this research is largely exploratory in nature. Three 
hypotheses were formulated: 

• HI: Audio feedback can better meet the conditions of ‘good’ formative assess¬ 
ment feedback, thereby constituting better feedback. 

• H2: Audio feedback can be more effective than written feedback in producing 
improvements in student learning and assessment scores. 

• H3: Audio feedback technology can better facilitate the efficient creation and 
delivery of feedback to students. 

Significant exploratory goals are an improved understanding of students’ use of audio 
feedback; for example, listening habits, how they use it, the role of portability, and 
whether students re-use audio feedback more so as to inform conclusions and future 
research. 


Participants 

The study participants were drawn from a first-year cohort studying the BA (Hons) 
Business Management and Information degree course. This business-oriented degree 
provides students with typical business skills, but is peculiar in its emphasis on tech¬ 
nology and information in business, particularly in areas pertaining to web technolo¬ 
gies, e-business, information systems and management. The study was conducted 
during semester one of the academic year 2009/10 for a module on web technologies. 

Twenty-four students agreed to participate in the study (Table 1). The small popu¬ 
lation available (n = 26) indicates that the sample used (n = 24) was large, representing 
92% of the entire cohort. Participants included male (n = 14) and female (n = 10) 
students, predominantly within the 18-24 age bracket (n = 22). The majority were 
registered for full-time study at the institution (n = 23). Excluding international 
students without such points, the mean UCAS point score of the sample upon begin¬ 
ning the degree course was 256 (standard deviation [SD] = 37), thus representing one 
of similar academic characteristics. As a general comment, it can be added that the 
participants began the degree course with a general aptitude for the business and infor¬ 
mation technology (IT) disciplines. 


Table 1. Demographic details of study participants. 




Male 

Female 



Total 

Student sample ( n = 24) 

n 

% 

n 

% 

n 

% 

Age 

18-24 

14 

58 

8 

34 

22 

92 

30-34 

0 

0 

1 

4 

1 

4 

40M4 

0 

0 

1 

4 

1 

4 

Total 

14 

58 

10 

42 

24 

100 

Study mode 

Full time 

14 

58 

9 

38 

23 

92 

Part time 

0 

0 

1 

4 

1 

8 

Total 

14 

58 

10 

42 

24 

100 
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‘Voice emails’ 

To deliver audio email feedback, Wimba Voice™ 6.0 was installed (Wimba 2009) 
within the virtual learning environment. Wimba Voice™ is a web-based tool capable 
of being bolted onto a variety of virtual learning environments and provides a series 
of audio tools such as a podcaster and voice enabled discussion fora. It also enables 
the creation and delivery of ‘voice emails’. These are essentially voice messages that 
can be recorded and communicated with students using a familiar email/tape-recorder 
interface, all within a Java-enabled web browser (as in Figure 1). It is also possible to 
send written emails with audio annotations. 

Using voice emails for this study was advantageous for several reasons. Firstly, the 
audio file is not attached to the email, thus obviating MP3 file size issues normally 
associated with email delivery (for example, Merry and Orsmond 2008; Rotheram 
2009). Audio is instead saved to a local server. The recipient of a voice email is 
provided with a hyperlink to follow and then offered the opportunity of streaming the 
audio within a web browser or downloading the message as an audio file. Recipients 
can listen or download the audio as many times as they wish. 

Secondly, the voice email enables students to reply with their own voice emails in 
much the same way that a reply might be sent to a conventional email. The importance 
ascribed to fostering a student-tutor dialogue in formative feedback has been well 
noted in theoretical work (Nicol and Macfarlane-Dick 2006) and corroborated more 
recently in audio feedback research (Fell 2009). The use of voice emails therefore 
contributes to a student-tutor dialogue in a way that other audio feedback delivery 
methods do not and its use here is an attempt to control for a known limitation of audio 
feedback. The majority of IT laboratories used by the cohort are equipped with head¬ 
set microphones as standard. 


Procedure 

The research was conducted with the student participants during the second half of 
semester one. The summative assessment for the course module required the submis¬ 
sion of an XHTML report. The module design was modified to incorporate a forma¬ 
tive assessment point mid-way through semester one. This entailed the submission of 
an XHTML report plan, thus providing tutors with feedback on student learning 
progress and understanding. 

To control for varying levels of student information and communications 
technology (ICT) efficacy, a pre-test orientation session with Wimba Voice™ was 
delivered to all students during the week preceding formative feedback delivery. 
This session covered how to access voice emails, download them and reply to them. 
A demonstration video was also created and posted on the relevant module section 
of the virtual learning environment. After submitting their formative assessment, 
students were then randomly streamed into two feedback groups: a written group 
(control) (n = 12); and a voice (email) group (treatment) (n = 12). Details are 
provided in Table 2. 

To minimise variability in the marking of the formative submission, module tutors 
agreed marking criteria and, where possible, attempted to incorporate aspects of Nicol 
and Macfarlane-Dick’s (2006) seven principles of good formative feedback. In line 
with formative feedback practice, no marks were attached to students’ formative 
assessment submissions; however, for the purposes of the current research, a mark 
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Figure 1. Creating a ‘voice email’ using Wimba Voice™. 


was recorded by tutors based on the agreed marking criteria. This mark remained 
undisclosed to students. 

Formative feedback was delivered to students within a week of submission. 
Students streamed into the treatment group received voice email feedback; students 
streamed into the control group received their feedback as an MS Word file email 
attachment. The required length of time taken to generate and deliver feedback was 
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Table 2. Details of participants in streamed feedback groups. 




Male 


Female 


Total 


n 

% 

n 

% 

n 

% 

Voice (n = 12) 

18-24 years old 

8 

66 

3 

25 

ii 

92 

30-34 years old 

0 

0 

1 

8 

i 

8 

Total 

12 

66 

4 

42 

12 

100 

Written (n = 12) 

18-24 years old 

6 

50 

5 

38 

23 

92 

40M4 years old 

0 

0 

1 

4 

1 

8 

Total 

14 

58 

10 

42 

24 

100 


recorded by tutors. This was measured from the moment the tutor began perusing the 
submission to the very end of feedback creation process (i.e. delivery to the student) 
so as to accommodate the total time that a tutor might invest in providing formative 
feedback. 

In the final week of the semester, students were required to submit their summative 
assessment (XHTML report). Summative assessment submissions were marked and 
written feedback delivered to all students. Student performance in the summative 
assessment was recorded for subsequent analysis. 


Research instruments 

Student participants in both the control and treatment groups received a web-based 
survey instrument designed to elicit data pertaining to feedback attitudes, initial use, 
reception and effect on learning. The survey was distributed to students one week after 
formative feedback was delivered and was administered during an IT laboratory 
session. The web-based survey consisted of three distinct sections. Section one was 
designed to capture simple demographic data (Table 1), while section three captured 
descriptive data on the extent of the formative feedback use, student ICT access and 
device ownership and use, and feedback preferences. 

To determine how well formative feedback achieved its purpose and to detect the 
effect of formative feedback on student learning, the design of section two of the web- 
based survey instrument was informed by Nicol and Macfarlane-Dick’s (2006) feed¬ 
back model and the feedback conditions proposed by Gibbs and Simpson (2004). 
Students were required to indicate their responses to a series of statements using a five- 
point Likert scale, ranging from ‘strongly agree’ (five) to ‘strongly disagree’ (one) (e.g. 
Table 3). These statements mapped to the above-noted models. 

Semi-structured interviews were conducted with a sample of the student partici¬ 
pants in the final week of the semester (n =10). These interviews were designed to 
gather rich data on audio feedback use, perceptions and to better understand the role 
of formative audio feedback on student learning. Interviews were administered by a 
member of the research team not involved in module teaching. Interviews were sound 
recorded, transcribed and then uploaded into QSR NVivo 8 for content analysis, 
coding and subsequent analysis. Coding was undertaken using Holsti’s (1969) meth¬ 
odologies for content analysis and category creation. 








Table 3. Measures of central tendency and Mann-Whitney U tests between groups for section two responses. 


46 G. Macgregor et al. 


N 


<D 

I 

GO 


Of 

vo 

<—i 


T—I 

On 

•n 

r- 

of 

00 

c-- 

CN 

CN 

Of 

of 

of 

I—H 

o 

oo 

of 

lO 


CO 

i—1 

of 

<n 

oo 

VO 

Of 

of 

of 

m 

or 

CN 

o 

m 

CN 

lO 

Of 


1—' 


Ov 

i-H 

CN 

Of 

© 

© 

© 

of 

of 

of 

t ; 

Os 

o 

© 

© 


o 

© 

© 

o 

o 

© 

© 

© 

o 

O 

O 

O 

o 

o 

CN 

of 

of 


ov 

o 

Ov 

VO 

(N 

VO 

^o 

o 


OO 

m 

Os 

of 



o 

m 


o 

VO 

VO 

i-H 

r- 

Cs) 

00 

CO 

CO 

of 

vo 

Os 

00 

CO 

o 

vo 

r- 


in 

in 

o 

of; 

o 

r- 

Os 

o 

o 


r- 

r- 

CO 

vo 

7 

o 

1 

© 

l 


7 

7 

o 

1 

7 

7 

o 

1 

7 

CN 

1 

CN 

1 

o 

1 

o 

1 

o 

l 

o 

1 

o 

1 

o 

o 

o 


o 

o 

o 

O 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

o 

lO 


in 

<n 

© 

<n 

o 

o 

o 

o 

o 

o 

o 

in 

o 

o 

o 

VO 

© 




t— H 


o 

t— H 

o 

t— H 

OS 

o 

1—H 


vd 

1—H 

vo 

VO 

vo 


oj- 

of 

r-~ 

<n 

VO 

vo 

of 

of 

CO 

vo 

VO 

VO 

vo 

r- 










•n 


in 

in 





>n 

Of 

Of 

T d‘ 



of 

of 

of 

of 

CO 

CO 

CN 

CO 

CO 

of 

of 

of 

cd 

lO 

r- 

m 



oo 

in 

<n 

<n 

CN 

r-~ 

r- 

00 

of 

m 

CN 

oo 

m 

CN 




Os 

o 

cq 

tq 

tq 

of 

i 

vq 

in 

cq 

r- 

Os 

in 

CN 

of 

of 

cd 


CO 

of 

CO 

CO 

CO 

CO 

CO 

ri 

CO 

CO 

CO 

CO 

cd 

cd 


















in 

of 

of 

of 


of 

of 

of 

of 

of 

of 

<N 

CN 

CO 

CO 

Of 

Of 

of 

cd 

00 

00 

CN 


oo 

00 

CN 

t-* 

CN 

OO 



CO 


(N 

OO 

(N 

CO 

o 

o 

Os 


in 

in 

os 

•—c 

Os 

in 

m 


00 


Os 

o 

of 

cq 


of 

cd 


CO 

CO 

CO 

of 

CO 

CO 

CN 

CN 

CN 

CO 

CO 

of 

cd 

cd 





o g 


G 

cd 

O 






bX) 




oi) 

cj 






U G 


|_| 

o 





G 








Jo 

g 


(D c/3 

-rt 

G 9 

o £ 


cd 

-G 

s 

O 

-4—< 





03 

G 




o 

cd 

X> 




tj 


.s ^ 


o 

C/3 

-4—< 

G 

cd 

> 

<D 

13 

S-l 





C/3 




03 

o 




<D 

fc 

o 

o 


<L) cd 

o w 


C/3 

■H 

£ 

M—< 





§ 

cd 

u 




,o 

G-h 

J-4 

o 



JD 

<+H 

C/3 

i-H 

o 

g 

03 

•b c 

=3 cd 


<L> 

Jo 

a 

C/3 

03 

§ 

Id 





3 

03 

O 

g 

o 

cd 


J) 

G 

G" 

> 

3 

’g 

o 


3 

rrt 


CD 

u ia 

t-i —t 


G 

O 

G 

O 




-4- 1 

<D 

G2 

rO 

03 


3 

> 

H 

o 



ffl P-h 


R. I prefer to receive my feedback electronically 3.67 4 3.58 4 60.50 -0.720 0.471 

Note: a Two-tailed Mann-Whitney U test (adjusted for ties), significant atp < 0.05. b Statement reverse coded. *p < 0.05. 
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Results 

Survey instrument 

Table 3 sets out the results from section two of the survey instrument. With such ordi¬ 
nal data it is conventional to consider median results, although mean results have been 
included for completeness. The largely favourable nature of responses indicates that 
both groups were generally satisfied with their feedback, whether it was in written or 
voice form. Notable median differences in group responses can be observed for state¬ 
ments I, J and K, indicating that students in the voice email group found their forma¬ 
tive feedback to better meet good formative feedback criteria in terms of detail and 
being understandable. A notable difference can also be found for statement L, reveal¬ 
ing that the voice email feedback failed to inspire and motivate students to the same 
degree as written feedback. A Mann-Whitney U test was conducted to detect signifi¬ 
cant differences between group responses for all questions (Table 3). Differences in 
results for questions J, K and L were noted as being statistically significant (p < 0.05), 
further corroborating some of the above-noted differences. It is worth noting that posi¬ 
tive mean responses for the voice email group can be observed for many of the 
question statements included in Table 3. 

Recall that section three of the survey instrument captured descriptive data on the 
extent of formative feedback use, student device ownership and use, and feedback 
preferences. No member of the voice group reported using the reply functionality of 
their voice emails to engage in a tutor-student dialogue, with 75% (n = 9) indicating 
that they found the feedback to be sufficiently clear and no clarification was required. 
The remaining 25% (n = 3) reported a preference for seeking clarification from a tutor 
in person, a result matched in the written group; however, only 25% (n = 3) of the 
written group reported finding the feedback to be sufficiently clear. Indeed, 17% (n = 
2) also replied in an email to ask further questions of their written feedback. One 
student failed to recall whether they had responded to their written feedback or not. 

Only 50% of the voice group (n = 6) reported saving their audio feedback to a 
device for subsequent listening or re-using it after delivery. Data captured by the 
survey instrument (Table 4) indicate a greater proclivity to re-use written feedback, 
rather than audio. It should, however, be noted that the survey instrument was admin¬ 
istered shortly after formative feedback was delivered and therefore failed to capture 
re-use behaviour in the week preceding summative assessment submission. 

Student participants reported wide ownership of a variety of mobile technologies 
capable of audio-file playback (Table 5); perhaps attributable to the ICT nature of the 
degree course they are studying. A significant proportion (25%) reported owning a 
smartphone (e.g. Blackberry™, iPhone™); however, far more owned a mobile phone 


Table 4. Re-use of formative feedback after delivery. 


Number of times re-used 

Voice (n = 

12) 

Written (n = 

12) 

n 

% 

n 

% 

0 

6 

50 

2 

17 

1 

2 

17 

1 

8 

2 

2 

17 

5 

42 

3 

1 

8 

1 

8 

>3 

1 

8 

2 

17 
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Table 5. Summary of mobile device ownership characteristics of student participants. 


Feedback group 

Smart phone 

Mobile with audio 

MP3 player 

Laptop 

n 

% 

n 

% 

n 

% 

n 

% 

Voice in = 12) 

4 

33 

8 

67 

9 

75 

ii 

92 

Written (n = 12) 

2 

17 

11 

92 

10 

83 

ii 

92 

Total ( n = 24) 

6 

25 

19 

79 

19 

79 

22 

92 


capable of MP3 playback (79%), an MP3 player (79%) (e.g. iPod™, Creative ZEN™) 
and laptop (92%). 


Academic performance data 

To evaluate students’ academic performance and measure the potential influence of 
formative voice email feedback on student learning, the academic performance of 
both groups was analysed. Students’ performance in the formative and summative 
assessments is set out in Table 6 and Figures 2 and 3. 

Student performance in the formative assessment was generally poor; although 
coincidentally both groups had an identical mean performance (Table 6). Performance 
in the summative assessment (after experimental treatment) was better, with the writ¬ 
ten group performing better, although an unpaired two-tailed t-test at p < 0.05 revealed 
no difference between group performances (f(22) = 0.43,/? = 0.67). It is nevertheless 
noteworthy that student performance in the voice group is less dispersed around the 
mean (SD = 9.5; R = 30). 

Whilst students’ performance in the formative assessment appears to be replicated 
in the summative assessment by the graph profdes provided in Figures 2 and 3, it is 
possible to observe learning gains in the performance of voice group participants in 
Figure 3 that are not observable in the written group. However, this observation does 
not appear to be borne out by the mean percentage learning gains achieved by students 
(Myoice = 24.34; /V/ wnttcn = 26.34) and was not corroborated by an unpaired two-tailed 
i-test at p < 0.05 on the individual learning gains achieved by student participants 
(t(22) = 0.53, p = 0.6). 


Table 6. Student performance in formative and summative assessments of written and voice 
groups. 



Voice ( n = 12) 

Written (n= 12) 

Formative performance measure 

Mean 

33.83 

33.83 

SD 

13.68 

12.36 

Range 

44 

36 

Summative performance measure 

Mean 

58.17 

60.17 

SD 

9.50 

12.78 

Range 

30 

43 

Mean percentage learning gain 

24.34 

26.34 
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Student performance (written) 



Student mark (formative) 


-Student mark (summative) 


Student (n = 12) 


Figure 2. Student performance in fonnative and summative assessment (written). 


Student performance (voice) 



Student mark (formative) 
Student mark (summative) 


Student (n = 12) 


Figure 3. Student performance in formative and summative assessment (voice). 


Interview data 

Iterative analyses of the interview data in QSR NVivo derived a hierarchical coding 
taxonomy (Table 7). This denotes the principal themes identified from the data. The 
taxonomy includes two super-ordinate categories (voice and written), each including 
a series of subordinate classes. The scope of these classes is delineated in Table 7 
along with indicative supporting quotes from the data, the number of interview 
sources in which the theme code was discussed, and the total references to this theme 
code across all interviews. Further discussion of these data is incorporated into the 
discussion section. 
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Table 7. Coding taxonomy derived from interview data. 

Category Total 

(A, B), classes Number of number of 

(A.l-B.l) Scope note Supporting quote sources references 


A. Voice 
A.l 

Dissatisfaction 


A.2 Portability 


A. 3 Voice- 
written 
combination 


A.4 Re-use of 
voice feedback 


Class 

encompassing 
feelings of 
dissatisfaction 
when using or 
receiving 
feedback in 
audio form. 


Comments or 
feelings 

pertaining to the 
portability (e.g. 
saving to 
multiple mobile 
devices for m- 
leaming) or 
flexibility 
afforded by 
receiving voice 
emails as a form 
of feedback. 

References to the 
perceived need 
to combine 
audio feedback 
with written 
feedback to 
improve 
feedback clarity 
and ease of use. 

References to the 
role of re-using 
feedback to 
improve 
learning. 


“I was quite impressed with it 
[...] I found it quite useful 
that you actually listen to 
someone and they sort of get 
their point across, rather than 
it being written down. 
Although, I did find that [...] 
when you’re in the library 
you can’t go and listen to it 
again, whereas you could 
have read it again. That’s like 
the only problem I’ve found. 
But other than that, it’s really 
good.” (Student 7) 

“I put it on a USB stick and 
listened to it at home. When I 
plug it in it goes straight to 
Windows Media Player and 
then it’s easy [...] With 
written every time I have to 
look at the papers for me to 
read but with the audio it’s 
like it’s straight on the 
computer ... you just log on 
and listen to it. So I don’t 
have to carry anything 
around or misplace it.” 
(Student 2) 

“Maybe like if you were given 
the audio file but also four or 
five main points in text about 
the audio file that would be 
good. But I suppose I could 
just write it down myself but 
I’m just being lazy really.” 
(Student 7) 

“I’ve received feedback three 
times now from different 
modules, and with written 
feedback I tend to access it 
once when I first get the 
email, and then normally just 
once more when I doing the 
actual report or assignment. 
So far, with the audio 
feedback I’ve listened to it a 
few times already and I 
haven’t even started the 
report.” (Student 3) 


1 


5 


4 


7 


1 


6 


5 


13 
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Table 7. ( Continued ). 


Category 

(A, B), classes 

(A.l-B.l) 

A.4.1 Referring 
back 


A.5 Satisfaction 


A.5.1 

Preference 


A. 5.1.1 Clarity 


Scope note 

Reference to 
difficulties in 
referring back to 
specific 
passages in 
feedback 
delivered. 

Passages noting an 
overall 

satisfaction or 
contentment 
with receiving 
formative 
feedback in an 
audio form. 

Explicit approval 
of audio above 
other feedback 
formats. 

Feedback was 
considered by 
the student to be 
clearer, easily 
understandable 
and more 
explicit that 
written 
feedback. 


A.5.1.2 Class denotes 

Detailed passages 

referring to the 
increased level 
of detail in 
audio feedback 
when compared 
with written 
feedback. 


A.5.1.3 Ease of References to the 
use level of 

difficulty in 
retrieving, 
listening, re¬ 
using or 
downloading 
feedback. Does 
not get lost; 
nothing to carry 
around 


Total 

Number of number of 
Supporting quote sources references 


“I thought it was very good and it 2 3 

helped me because it was 
specific to my needs. The only 
one problem I found with it is 
that it is hard to refer back to, 
where you can with a written 
thing. But I definitely prefer 
audio than written.” (Student 7) 

“I was quite impressed with it. I 3 4 

found it quite useful that you 
actually listen to someone and 
they sort of get their point 
across, rather than it being 
written down.” (Student 7) 


“I prefer the audio feedback over 6 8 

written feedback.” (Student 1) 


“I thought he put the message 7 12 

through very clearly and 
concisely. He spoke slowly 
enough to be able to understand 
it easily. But also not so it came 
across as too slow and jokey, if 
you know what I mean? There 
was more emphasis on certain 
words which you don’t get from 
the written feedback, so you 
know... And also probably the 
order of the feedback may have 
been in order of relevance for 
you to look at.. .1 think as well.” 

(Student 3) 

“Oh yeah, it was well detailed. It 8 10 

was 3 mins long of what I 
should have done, how I could 
have made it better so it was 
definitely more detailed than 
the written feedback.” (Student 
1 ) 


“It was dead easy to access it 7 8 

straight away [...] Yeah, I 
saved it to my files at home 
yeah.” (Student 1) 
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Table 7. ( Continued ). 


Category 

(A, B), classes Number of Totalnumber 

(A.l-B.l) Scope note Supporting quote sources of references 


A.5.1.4 

Engagement 


A.5.1.5 Face- 
to-face 
emulation 


Student 
perceptions of 
audio feedback 
as exciting and/ 
or engaging. 

Passages in which 
the student 
referred to the 
ability of audio 
to replicate or 
emulate normal 
face-to-face 
meetings with 
the module 
tutor. 


A.5.1.6 Denotes passages 

Personal in which the 

nature student notes or 

describes the 
benefits of 
audio feedback 
as the 

informality, 
personalisation 
or relaxed tone 
of the 
feedback. 


A.5.1.7 References to the 

Retention role of audio 
feedback in 
aiding retention 
of important 
points or issues 
raised in the 
feedback 
received. 


“For me, I think it was quite 
exciting, because it’s like just 
talking to [the tutor] and 
listening to him. I think it’s 
good.” (Student 2) 

“With the audio feedback it was 
a bit ... it was nothing I’d 
ever experienced before 
except face to face with a 
teacher, so I just rewound 
and re-listened to the 
feedback without taking any 
notes on it.” (Student 4) 


“I quite liked it. I liked the way 
that it felt more personal to 
you, even though all 
feedback should be personal 
to you because obviously it’s 
the tutor you know, and 
you’ve worked with before. 
And the fact he’s actually 
saying your name rather than 
... you could perceive he’s 
just copying and pasting the 
same feedback. I’m sure it 
wouldn’t happen but it could 
be perceived that way. I like 
the way it seemed more 
personal and I certainly felt 
that what [the tutor] said 
about my report plan was 
pretty accurate. And then that 
spurred me on to actually 
make the changes that he 
suggested.” (Student 3) 

“It was better than sitting there 
and reading, when you’re 
listening to it you take in 
more than looking at it, and 
just throw it aside.” (Student 

5) 


A.5.1.8 Time Denotes reference 

saving to the ability of 

audio feedback 
to save time on 
the part of the 
student, or 
related issues. 


'I think it was more detailed, 
telling me everything I have 
to do. So when I listen to it, I 
think it saves time.” (Student 
2 ) 


3 


2 


6 


3 


1 


3 


4 


11 


4 


1 
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Table 7. ( Continued ). 


Category 

(A, B), classes Number of Total number 

(A.l-B.l) Scope note Supporting quote sources of references 


A.5.1.9 Voice 
characteristi 
cs 


B Written 


Passages 

discussing or 
referring to the 
positive role of 
voice tone, 
intonation or 
characteristics 
in the feedback 
students 
received. 


“Like I said about [the the 
tutor], it’s similar to how he 
speaks in the lectures, so I’m 
more used to that way of him 
speaking. You know, he 
could be sounding like a 
completely different person 
in written feedback. I think 
he can explain more, expand 
more when he’s talking 
about it rather than in an 
email - he may find that 
difficult.” (Student 6) 


B.l Preference 


“Probably written, because it’s 
there in front of me when 
I’m doing my work. 
Whereas the audio you have 
to sit at the computer and 
listen to it.” (Student 8) 


3 


3 


4 


5 


Comparison of time requirements 

To minimise the influence of varying voice characteristics and intonation on students’ 
perception of voice email feedback, all audio feedback was delivered by a single 
member of the teaching team. To enable comparisons to be made between the time 
taken to deliver audio and written feedback created by different tutors, a sample of 
submissions (n =12) was taken from another cohort to provide ‘dummy’ feedback for 
benchmarking purposes (Table 8). Data are provided in decimal time and minutes/ 
seconds. The time taken for both tutors to complete the dummy feedback for the same 
submissions was generally similar and did not differ significantly for either voice 
emails (t(10) = 1.52 ,p = 0.16) or written feedback (t(10) = -0.61 ,p = 0.56). 

The time requirements for generating voice email and written feedback for the 
experimental sample are provided in Table 9. The time requirements for voice email 
feedback were significantly smaller and reveal that audio was almost twice as fast as 
written feedback. Using audio also appears to promote less variability in the amount 
of marking time spent per student submission, as indicated by the reduced data disper¬ 
sion (i.e. SD and R ). 

Discussion 

This study aimed to investigate and evaluate the efficacy of audio technologies in 
delivering formative feedback, its ability to meet recognised formative feedback 
models, and explored its influence on student learning. Despite performing similarly 
in the formative assessment and indications from the survey instrument and interview 
data that voice email better achieved certain conditions of ‘good’ formative feedback 
(supporting our first hypothesis), the written group actually performed better in the 
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Table 8. Benchmark timings for delivery of ‘dummy’ audio and written feedback. 



Tutor A 


Tutor B 


Dummy 

feedback 

Minutes/seconds 

Decimal 

Minutes/seconds 

Decimal 

Voice 

1 

4.49 

0.08028 

3.35 

0.05972 

2 

4.49 

0.06667 

3.42 

0.06167 

3 

4 

0.06722 

4.12 

0.07 

4 

4.02 

0.07889 

4.24 

0.07334 

5 

4.44 

0.06833 

4.05 

0.06806 

6 

4.06 

0.06694 

3.54 

0.065 

Mean 

4.17 

0.07138 

3.58 

0.06629 

SD 

0.22 

0.00638 

0.18 

0.00515 

Total 

25.41 

0.42833 

23.51 

0.39778 

Written 

7 

7.59 

0.13306 

8.27 

0.14083 

8 

7.36 

0.12667 

7.16 

0.12111 

9 

7.13 

0.12028 

7.42 

0.12833 

10 

7.01 

0.11694 

7.27 

0.12416 

11 

5.5 

0.09722 

6.13 

0.10361 

12 

5.12 

0.08667 

5.54 

0.09833 

Mean 

6.48 

0.11347 

7.09 

0.11939 

SD 

1.04 

0.01788 

.57 

0.01586 

Total 

40.50 

0.68083 

42.59 

0.71639 


Note: Decimal values to five decimal points. 


summative assessment, although this difference was not significant. Interview partic¬ 
ipants noted an increased desire to re-use audio as opposed to written feedback and 
revealed a clear preference for audio owing to the fact that it was easier to understand, 
more detailed and personal. The results from our analysis of students’ summative 
assessment performance were therefore not intuitively anticipated and were disap¬ 
pointing. Whilst the generally high level of data variability and small participant 
numbers may have contributed to this finding, we nevertheless have to reject our 
second hypothesis in this instance. However, the large time efficiencies recorded in 
our study verifies our third hypothesis and helps to clarify conflicting evidence on the 


Table 9. Time requirements for delivering voice and written feedback. 




Mean 


SD 


Range 

Feedback 

group 

Decimal 

Minutes/seconds 

Decimal 

Minutes/seconds 

Decimal 

Minutes/seconds 

Voice 

0.07574 

4.32 

0.01474 

0.53 

0.04944 

2.57 

Written 

0.13498 

8.50 

0.03794 

2.16 

0.13250 

7.57 


Note: Decimal values to five decimal points. 
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time requirements for generating audio feedback. This is an encouraging finding as it 
supports the view that audio feedback provides improved opportunities for adhering 
to good pedagogical practice by better enabling the formative assessment to be 
embedded within curriculum design. 

The survey instrument elicited generally positive data in favour of audio feedback, 
some of which was statistically significant and much of which was corroborated by 
interview data (A.5 and subclasses). In particular, students found the audio feedback 
to be clearer and easier to understand and interpret. Literature on assessment feedback 
notes that it can often be difficult for students to interpret and decode feedback, owing 
to the use of language and jargon (e.g. “This is insufficiently critical”). It is suggested 
that this ‘information transmission’ approach to feedback delivery causes confusion in 
learners as they are consequently unsure how best to correct their learning behaviour 
(Nicol and Macfarlane-Dick 2006). It is encouraging that audio was more effective in 
providing clear and understandable feedback and this provides an indication that audio 
feedback could positively enhance the student learning experience. Interview data 
(coded at A.5.1.1 - Clarity) tend to support this observation and reveal the increased 
detail of audio feedback as being the facilitator of such clarity: 

I thought [the tutor] put the message through very clearly and concisely. He spoke slowly 
enough to be able to understand it easily. But also not so it came across as too slow and 
jokey, if you know what I mean? There was more emphasis on certain words which you 
don’t get from the written feedback, so you know ... And also probably the order of the 
feedback may have been in order of relevance for you to look at ... I think as well. 
(Student 3) 

I thought it was better than getting written feedback because sometimes you can’t read 
people’s writing either, whereas with the listening one you can understand it a bit more. 

I thought it was quite detailed. Because he said that I should put in some other bits 
because there was too many parts and stuff, and that I should put in other bits. And other 
bits that I should think about writing and stuff. I found it dead good [...] It’s more 
modern isn’t it. Because sometimes people have got dead scribbly handwriting and then 
you’ve got to go back to them, and ask “what does this say? I can’t read it”. Whereas, 
with the listening one it’s easier to understand unless they have a really strong accent, 
but I could listen to mine perfectly. (Student 5) 

A lot of the time with written feedback I can get confused with what they actually mean 
by it, whereas with audio I think they come across more understanding. I think maybe 
by saying it they can explain it better than writing it sometimes. I find that useful to 
gather the feedback. A lot more detailed than written feedback. I think when they say it 
they can understand what they have just said may not have made sense, and they can 
expand on it more. (Student 6) 

Nevertheless, statistically significant differences in group responses for statement 
L in the survey instrument (“The feedback helped to increase my interest in the 
module I am studying”) also revealed that the potential of audio feedback to better 
engender interest and motivation among students in their subject of study may have 
been overstated by previous research (Cryer and Kaikumba 1987; Ice et al. 2007). The 
results from this study suggest that written feedback better inspired motivational 
beliefs in student participants. Similarly, inspiring motivation and interest in the topic 
of study were not themes that emerged strongly from the interview data. Students 
often reported positively on the personal nature of the feedback (A.5.1.6) and its abil¬ 
ity to emulate face-to-face interactions (A.5.1.5), but few associated this with 
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improvements in enthusiasm. Some also reported feelings of engagement (A.5.1.4) 
and “excitement”; but this appears to have been attributable more to the novelty of 
receiving audio feedback rather than an inherent ability of recorded speech to be 
engaging. 

As noted above, the data indicate an overall preference for audio feedback, which 
reflects the findings of existing research and was anticipated. It is nevertheless note¬ 
worthy that although the survey and interview data are largely supportive of this view, 
data from the latter uncovered several students who expressed a preference for written 
feedback (B.l). Of the students that explicitly expressed a preference for audio feed¬ 
back, four also articulated a desire to receive their feedback in both audio and written 
formats in future (A.3). Interrogation of the qualitative data indicates that two students 
identified issues relating to “referring back” (A.4.1) as motivating this view. Referring 
back to specific passages of feedback or refreshing on its key points is certainly 
simpler to do with written feedback, either from scanning or searching the text. This 
view links with the findings of Cryer and Kaikumba (1987) and was further corrobo¬ 
rated by Students 4 and 8, who stated: 

[0]ne thing that I prefer about the written feedback is where I can make a list and tick 
stuff off ... improve my work as I go along and check what I’ve done. 

[Written feedback can be useful] because it’s there in front of me when I’m doing my 
work. Whereas the audio you have to sit at the computer and listen to it. 

Even students favourably disposed to audio feedback remarked on the lack of flexibil¬ 
ity sometimes afforded by audio approaches: 

I was quite impressed with it [...] I found it quite useful that you actually listen to some¬ 
one and they sort of get their point across, rather than it being written down. Although, 

I did find that [...] when you’re in the library you can’t go and listen to it again, whereas 
you could have read it again. That’s, like, the only problem I’ve found. But other than 
that, it’s really good. (Student 7) 

Interestingly, many students stated a preference for not downloading audio feed¬ 
back to another device and instead preferred streaming the audio, an unusual finding 
given the wide ownership of mobile devices by student participants (Table 5) and the 
increased study flexibility that this would have afforded students. Instead, numerous 
students noted that they preferred to leave the voice email within their email software 
and revisit when necessary. Students 4 and 6 are indicative: 

No, I didn’t download the file. [...] Not sure if I would have. I just felt I could rewind it 
if I wanted to and could go back to my emails if I needed to. 

I have not put it [the audio file] on my MP3 player or anything like that. I’ve just listened 
to it straight off the emails. 

Even those students that saved the file did so to their laptop or to the university 
networked drive, and not to a mobile device. As Students 3 and 2 noted: 

I initially listened to it through streaming because I wasn’t at my own computer at the 
time. So, I didn’t have anywhere to save it to. Since then I’ve saved it to my laptop so I 
can use it for easier access. 
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I put it on a USB stick and listened to it at home. When I plug it in it goes straight to 
Windows Media Player and then it’s easy [...] With written [feedback] every time I have 
to [find] the papers to read but with the audio it’s like it’s straight on the computer ... 
you just log on and listen to it. So I don’t have to carry anything around or misplace it. 

Although viewed positively by students and enabling a level of portability (A.2), such 
‘file saving behaviour’ immediately places restrictions on the level of access possible 
within particular learning contexts, as Student 7 notes above (e.g. at the library or IT 
laboratory), and for this reason remains much the same as relying on the streaming of 
audio via email. Promoting m-leaming is a motivation behind using the audio feed¬ 
back format, and the expectation is that students will in most cases save the file to a 
mobile device, thus enabling quick and flexible access to their feedback. This finding 
may therefore appear to constitute strange personal information management (PIM) 
behaviour on the part of our student participants. 

Email has been found to be an important PIM tool in the personal archiving of docu¬ 
ments and in task management (Whittaker, Bellotti, and Gwizdlca 2006), and studies 
within information science note that an increasingly popular strategy in PIM is actually 
“to do nothing” (Bruce, Jones, and Dutnais 2004). This is normally because saving and 
filing information can prove cognitively onerous and, when required to re-find personal 
information, most users are often successful anyway. It could therefore be suggested 
that our student participants chose in many cases not to save the fde for fear of not 
being able to re-find it at a later date and instead preferred to do nothing, where at least 
the voice email could always be re-found (via advanced searching or file browsing) 
for future study tasks. A potential contributory factor explaining this PIM behaviour 
may be related to the clarity and detail of the audio feedback, which may have been 
such that it reduced - at least initially - students’ perceived need to revisit it at a future 
date. ‘Doing nothing’ may therefore have been considered by many participants as the 
most appropriate form of behaviour. Feedback clarity also appears to have usurped 
students’ need to enter into a student-tutor dialogue about their feedback. 

Although only a small number reported re-using their audio feedback in the survey 
instrument, interview data indicated that more students re-used their audio feedback 
as the summative assessment deadline approached, as the comments of Student 3 
illustrate: 

I’ve received feedback three times now from different modules, and with written feed¬ 
back I tend to access it once when I first get the email, and then normally just once more 
when I am doing the actual report or assignment. So far, with the audio feedback I’ve 
listened to it a few times already and I even haven’t started the report. 

This increased level of audio re-use is consistent with the findings of Sipple (2007) 
and Ice et al. (2007) who, in different ways, suggest that increased feedback use will 
result in greater feedback application in learning tasks that, in turn, will stimulate 
learning gains among students. Our findings on academic performance and group 
learning gains do not appear to corroborate their assumptions in this instance. It is 
worth noting that Ice et al. (2007) also attempted to examine the degree to which feed¬ 
back was applied by students in future work. 

Conclusions 

This exploratory study aimed to investigate and evaluate the efficacy of audio tech¬ 
nologies (voice email) in delivering formative feedback and its influence on student 
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learning. It was motivated by the potential for time efficiencies in feedback delivery 
and as a potential solution to the lack of formative learning at higher education. In 
particular, our research was motivated by work hypothesising that audio feedback could 
be capable of improving student learning. Our results tend to support the view that audio 
feedback can be more efficient and better meets existing models of ‘quality’ or ‘good’ 
formative feedback, as posited by the literature, thus enhancing the student learning 
experience and better informing strategic policies with respect to assessment practice 
at higher education. Qualitative data provided useful insights into students’ perceptions 
of audio feedback and has improved our understanding of how it is used and re-used 
by students. However, despite better conforming to formative feedback models and 
high levels of feedback re-use by student participants, we found no significant differ¬ 
ences between groups in either the formative or summative assessment tasks. 

Since the study is exploratory in nature, only a small number of participants were 
used. This is a clear limitation of the study design, and the resultant findings are there¬ 
fore intended to be indicative rather than generalisable. Neither can we discount the 
possibility that students failed to engage with the assessment task at the level required, 
instead adopting a ‘surface learning’ approach. Future research should therefore seek 
to better understand the level of student engagement prior to formative assessment 
submission and study the learning behaviour of students in the weeks preceding 
summative assessment submission. Our interview data provide a useful basis for 
further qualitative work, providing a taxonomy of factors that influence audio feed¬ 
back effectiveness. Future research should employ further qualitative techniques to 
better understand the varying nature of the feedback content delivered using audio and 
written methods and monitor resultant student learning. This would inform the devel¬ 
opment of a theoretical model to better guide assessment practice and maximise the 
effectiveness of audio feedback by practitioners. Replicating aspects of the study 
using larger cohorts is recommended to corroborate the quantitative findings derived 
from the survey instrument and from observing academic performance. 
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